Skip to content

scaleset: fix reapTimedOutRunners deadlock on creating instances#814

Merged
gabriel-samfira merged 1 commit into
cloudbase:mainfrom
nexthop-ai:fix/scaleset-reap-creating-deadlock
Jul 7, 2026
Merged

scaleset: fix reapTimedOutRunners deadlock on creating instances#814
gabriel-samfira merged 1 commit into
cloudbase:mainfrom
nexthop-ai:fix/scaleset-reap-creating-deadlock

Conversation

@benoit-nexthop

@benoit-nexthop benoit-nexthop commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

reapTimedOutRunners skipped runners in pending_delete/pending_force_delete/deleting/deleted status, but not creating/pending_create. When a runner's scale-set-level bootstrap timeout elapsed while its instance was still being created in the provider (e.g. a cloud provider deploy running long due to host iowait) and it had never joined GitHub, this tried to transition the instance straight from creating to pending_delete. That transition isn't allowed (InstanceCreating only permits -> error or -> running), so UpdateInstance returned a BadRequestError.

That error was treated as fatal: reapTimedOutRunners immediately unlocked everything it had processed so far and returned early, and its caller (consolidateRunnerState) aborted the rest of consolidation for that scale set on any error. This meant a single runner still legitimately booting (just slow) could block reaping of every other eligible runner in the same scale set, repeating every consolidate tick (~5 min) until the poisoned runner resolved on its own via the provider worker's separate create-timeout handling.

Observed in our prod cluster: invalid instance status transition from creating to pending_delete logged repeatedly for a scale set while the underlying cloud provider was slow.

Fix:

  • Skip InstanceCreating/InstancePendingCreate in reapTimedOutRunners, mirroring the same skip already done later in consolidateRunnerState. The provider worker owns create timeouts/failures for these instances.
  • Don't let one runner's transition error abort the whole batch: log it, release just that runner's lock, and keep processing the rest instead of returning early.

reapTimedOutRunners skipped runners in pending_delete/pending_force_delete/
deleting/deleted status, but not creating/pending_create. When a runner's
scale-set-level bootstrap timeout elapsed while its instance was still
being created in the provider (e.g. CloudStack deploy running long due to
host iowait) and it had never joined github, this tried to transition the
instance straight from creating to pending_delete. That transition isn't
allowed (InstanceCreating only permits -> error or -> running), so
UpdateInstance returned a BadRequestError.

That error was treated as fatal: reapTimedOutRunners immediately unlocked
everything it had processed so far and returned early, and its caller
(consolidateRunnerState) aborted the rest of consolidation for that scale
set on any error. This meant a single runner still legitimately booting
(just slow) could block reaping of every other eligible runner in the same
scale set, repeating every consolidate tick (~5 min) until the poisoned
runner resolved on its own via the provider worker's separate create-
timeout handling. Observed in prod: 'invalid instance status transition
from creating to pending_delete' logged repeatedly for the micro scale set
while CloudStack was slow due to iowait.

Fix:
  * Skip InstanceCreating/InstancePendingCreate in reapTimedOutRunners,
    mirroring the same skip already done later in consolidateRunnerState.
    The provider worker owns create timeouts/failures for these instances.
  * Don't let one runner's transition error abort the whole batch: log it,
    release just that runner's lock, and keep processing the rest instead
    of returning early.
@benoit-nexthop benoit-nexthop marked this pull request as ready for review July 7, 2026 12:55

@gabriel-samfira gabriel-samfira left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thanks!

@gabriel-samfira gabriel-samfira merged commit 9f4309d into cloudbase:main Jul 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants